System and method for efficient surface measurement using a laser displacement sensor

ABSTRACT

This invention provides a system for measuring displacement of an object surface having a displacement sensor that projects a line on the object surface and receives light from the projected line at an imager in a manner defines a plurality of displacement values in a height direction. A vision system processor operates on rows of imager pixels to determine a laser line center in columns of imager pixels in each of a plurality of regions of interest. Each region of interest defines a plurality of rows that correspond with expected locations of the projected line on the object surface. A GUI can be used to establish the regions. In further embodiments, the system generates grayscale images with the imager. These grayscale images can be compared to a generated height image to compensate for contrast-induced false height readings. Imager pixels can be compared to a reference voltage to locate the line.

RELATED APPLICATIONS

This application is a continuation of co-pending U.S. patent applicationSer. No. 15/456,676, filed Mar. 13, 2017, entitled SYSTEM AND METHOD FOREFFICIENT SURFACE MEASUREMENT USING A LASER DISPLACEMENT SENSOR whichapplication is a divisional of U.S. patent application Ser. No.14/149,774, filed Jan. 7, 2014, issued as U.S. Pat. No. 9,605,950 onMar. 28, 2017, entitled SYSTEM AND METHOD FOR EFFICIENT SURFACEMEASUREMENT USING A LASER DISPLACEMENT SENSOR, which application claimsthe benefit of U.S. Provisional Application Ser. No. 61/826,416, filedMay 22, 2013, entitled SYSTEM AND METHOD FOR EFFICIENT SURFACEMEASUREMENT USING A LASER DISPLACEMENT SENSOR, the entire disclosure ofeach of which applications is herein incorporated by reference.

FIELD OF THE INVENTION

This invention relates to machine vision systems used in determining thedisplacement of an imaged object surface in three dimensions (3D).

BACKGROUND OF THE INVENTION

In manufacturing and assembly processes, it is often desirable tomeasure some or all of an object surface with a high degree of accuracyand generate a map of the overall displacement or “profile” (e.g. heightin the physical z-coordinate direction) with respect to variouslocations on the object surface. This profile can be determined using amachine vision system (also termed herein “vision system”) in the formof a laser displacement sensor (also termed a laser beam “profiler”). Alaser displacement sensor captures and determines the (threedimensional) profile of a scanned object surface using a planar curtainformed by optically spreading a laser beam in a “fan” transverse to thebeam propagation path. In a conventional arrangement, a vision systemcamera assembly is oriented to view the plane of the beam from outsidethe plane. This arrangement captures the profile of the projected line(e.g. extending along the physical x-axis) on the object surface, which,due to the baseline (i.e. the relative spacing along the y-axis) betweenthe beam (fan) plane and the camera causes the imaged line to appear asvarying in the image y-axis direction as a function of the physicalz-axis height of the imaged point (along the image x-axis). In a typicalarrangement the camera optical axis intersects the laser plane at anacute angle and the well-known Scheimpflug configuration of the laserplane, camera lens, and camera image sensor can be used to form an imagein which the laser beam deviation is in focus through the region ofinterest despite the varying distance from the camera. This deviationrepresents the height profile of the surface. Laser displacement sensorsare useful in a wide range of inspection and manufacturing operationswhere the user desires to measure and characterize surface details of ascanned object via triangulation. One form of laser displacement sensoruses a vision system camera having a lens assembly and image sensor (or“imager”) that can be based upon a CCD or CMOS design. The imagerdefines a predetermined field of grayscale or color-sensing pixels on animage plane that receives focused light from an imaged scene through alens.

In a typical arrangement, the displacement sensor and/or object are inrelative motion (usually in the physical y-coordinate direction) so thatthe object surface is scanned by the camera, and a sequence of imagesare acquired of the laser line at desired spatial intervals—typically inassociation with an encoder or other motion measurement device (or,alternatively, at time based intervals). Each of these single profilelines is typically derived from a single acquired image of a largerfield of view that is known to contain the projected line. These linescollectively describe the surface of the imaged object. The field ofview is characterized by a working distance—that is, the surface of theobject should reside between a maximum and minimum height toappropriately capture profile information. Within this working distance,the size and shape of the line can vary based upon a variety of factors,including the orientation and reflectivity of the surface being scanned,the varying thickness of the laser plane (the line is typicallynarrowest at a “waist” at some intermediate depth from the sensor, andwider both closer to and farther from the sensor), and the varyingamount of magnification and foreshortening in the camera system as afunction of height. This variation in line size/geometry poses one ofseveral challenges in obtaining an accurate measurement of a surface ata desired scanning speed.

In measuring an object surface profile, it is sometimes desirable tosimultaneously generate a grayscale image of the object. While suchgrayscale images can provide a view of the object, the grayscale imagesmay have differing characteristics that can prevent a useful comparisonbetween it and the derived object surface profile. For example, thegrayscale image and other images generated may have different pixelscales. In this regard, if an object in physical space is further from acamera, e.g., at a lower height, then a pixel can appear wider and covermore physical space than an object closer to the camera. This effect cancause distortion in the resulting grayscale image and can prevent ormake difficult a comparison between the grayscale image and other imagesgenerated during the process. It is therefore desirable for thegrayscale images to have uniform pixel scales, both within a singlegrayscale image and among separate grayscale images.

Machine vision applications often require the use of the highest qualityimaging components. In recent times, the cost of imaging equipment, suchas cameras, has increased. As a result, those in the market may beforced to choose between paying the ever-increasing costs for thehighest quality of components or to accept lesser quality imagingequipment at a lower cost. Such lesser quality equipment often includesdefects or irregularities that make it less desirable for use in thanthe use higher quality equipment. The use of such lesser qualityequipment can potentially degrade the results of machine visionapplications. In particular, sensors can often have one or more badsensing elements. Bad sensing elements may be “stuck” in that they donot respond significantly to illumination at all, but show anapproximately constant value. Other bad sensing elements may respond toillumination, but with one or more parameters of the response, such asoffset, gain, nonlinearity, or noise level being significantly differentfrom the nominal response of “good” elements.

SUMMARY OF THE INVENTION

This invention overcomes disadvantages of the prior art, in illustrativeembodiment by providing a laser displacement sensor system, andcorresponding methods for scanning the height profile of an objectsurface, which increases scanning throughput, processing speed,efficiency and versatility of the system. In an illustrative embodiment,a displacement sensor projects a line on the object surface and receiveslight from the projected line at an imager in a manner defines aplurality of displacement values in a height direction. A vision systemprocessor operates on rows of imager pixels to determine a laser linecenter, or more generally, the position of the laser line and/orspecified portions of the line (e.g. the stop and start points of theline thickness) in columns of imager pixels in each of a plurality ofregions of interest. Rows in areas outside the regions of interest aretypically not acquired and/or processed so that the overall speed ofacquiring and processing an image of the line is increased. Each regionof interest defines a plurality of rows that correspond with expectedlocations of the projected line on the object surface. A height image isgenerated from the locations of the projected laser line in each scannedimage as the object moves relative to the imager field of view. A GUIcan be used to establish the regions, in which the user enters numerical(physical unit) or graphical boundaries to each region. In furtherembodiments, the system uses line width, intensity, and position togenerate grayscale information (e.g. along each laser line of the scan),or full grayscale images, with the imager and laser illumination. Thegrayscale image data and height image data can be used in conjunctionwith each other to generate correction factors that compensate forcontrast-induced false height readings. Illustratively, imager pixelscan also be transmitted to the processor in a “binary” manner bycomparing each pixel intensity (“intensity” defined as including, butnot limited to, a voltage value) in each column of the imager directlyto a reference intensity. Those pixels that meet or exceed the referenceintensity are provided with a logical 1 (or other) value and those thatfall below the threshold are provided with a logical 0 (or other) value.The line “center” is determined in each column from a likely grouping of“1” pixels, and that location is forwarded to the processor to form partof the height image. Illustratively, the center can be a physical centerof the line or any other convenient specified location associated withthe line, such as the start and end points.

In an illustrative embodiment, a system and method for measuringdisplacement of an object surface includes a displacement sensor thatprojects a line on the object surface and receives light from theprojected line at an imager in a manner that defines a plurality ofdisplacement values in a height direction. A vision system processoroperates to process rows of imager pixels to determine a position of theprojected line in columns of the imager pixels exclusively in each of aplurality of regions of interest. Each of the regions of interestrespectively defines a plurality of rows of imager pixels thatcorrespond with expected locations of the projected line on the objectsurface. A location of the position of the projected line in each of thecolumns corresponds to a physical displacement value on the objectsurface. Illustratively, the object and the displacement sensor are inrelative motion, and a plurality of acquired images are acquired of theprojected line. The acquired images define an overall height image ofthe object surface. A graphical user interface (GUI) is operativelyconnected to the vision system processor. It enables a user to enterboundaries of each of the regions of interest. The GUI provides inputsfor numerical physical units of the boundaries. Illustratively, the GUIcan display an acquired image of the object with the projected linecontained therein. This GUI is constructed and arranged to allowgraphical positioning of the boundaries by the user. The acquired imageis movable in the GUI to determine a range of displacement of theprojected line during a scan. In embodiments, the position of theprojected line is located at an approximate center along a columndirection. In embodiments, a measurement process analyzes acquired imagedata in each of the plurality of regions of interest to generateinformation for use in a downstream processing task. The informationused in the downstream processing task is generated from a combinationof analyzed acquired image data from the plurality of regions ofinterest. The downstream processing task can employ information from atleast one of a single acquired image of the object (a slice), aplurality of acquired images of the object (a group of slices), and allof the acquired images corresponding to the object (an overall heightimage). The downstream processing task can comprise a vision system taskor tool of custom or conventional design. The vision system task or toolcan comprise at least one of a registration, inspection, alignment andpattern-matching. Other tasks/tools can include smoothing, averaging,other image processing operations, etc., which can be performed, forexample, on a group of slices. Illustratively, the downstream processingtask includes a decision-making task. This decision-making task can beconstructed and arranged to perform at least one of an object rejection,line control or alert process. The measurement process can employ asingle scan of the object or a predetermined number of scans in a group(such as in smoothing, averaging, other image processing operations,

In another illustrative embodiment, a system and method for measuringdisplacement of an object surface includes a displacement sensor thatprojects a line on the object surface and receives light from theprojected line at an imager in a manner defines a plurality ofdisplacement values in a height direction. A vision system processoroperates to process rows of imager pixels to determine a position of theprojected line in columns of the imager pixels. The vision systemprocessor is constructed and arranged to generate, from the position ofthe projected line, a height image of the object surface. The visionsystem processor is also constructed and arranged to generate, from theintensity values of the projected line, a grayscale image of the objectsurface. The vision system processor is constructed and arranged toemploy measurements in the grayscale image to compute and applycorrection factors to the height image so as to compensate for errors inthe height image. The errors can comprise false height readings in theheight image, such as encountered at a contrast change on thesurface—for example where a printed character or symbol is provided in acontrasting shade or color. In embodiments, a measurement processemploys (a) measurements of data from the height image to constructmeasurements of the grayscale image, or (b) measurements of thegrayscale image to construct measurements of the height image. Adecision-making process can also be provided, which generates decisionsbased upon the measurement process.

In yet another illustrative embodiment, a system for measuringdisplacement of an object surface comprises a displacement sensor thatprojects a line on the object surface and receives light from theprojected line at an imager in a manner that defines a plurality ofdisplacement values in a height direction. A threshold circuit receivesrespective intensity values from pixels of the imager and performs acomparison of the respective intensity values to a reference intensity.A binary value is thereby assigned to each of the pixels based upon thecomparison. A vision system processor operates to process rows of thepixels to determine a position of the projected line in columns of thepixels having a binary value indicating presence of the projected line,the vision system processor being constructed and arranged to generate aheight image from the position of the projected line in the columns ofthe pixels. Illustratively, the intensity is defined as one of either avoltage or a current.

In yet another illustrative embodiment, a system for correcting asuspect pixel in a grayscale image, comprises a processor and a memoryincluding instructions thereon that, when executed by the processor,cause the processor to generate laser line position data andcorresponding laser line intensity data based in part on an acquiredimage of an object, where the laser line position data comprises atleast one laser line position datum. The processor can identify at leastone laser line position datum and substitute, using the processor, anoriginal value of the laser line intensity datum corresponding to theidentified laser line position datum with a substitute value for thelaser line intensity datum. The processor can generate the grayscaleimage from the laser line position data and corresponding substitutedlaser line intensity data.

In yet another illustrative embodiment, a system for generating auniformly scaled grayscale image corresponding to a height imagecomprises a processor and a memory including instructions thereon that,when executed by the processor, cause the processor to generate laserline position data and laser line intensity data, each pixel of thelaser line intensity data including an indicia corresponding to ameasured intensity. The processor can map each pixel of the laser lineposition data into a coordinate system representing a physical space, atleast two of the mapped pixels having a connection feature disposedtherebetween, the connection feature including indicia corresponding tothe measured intensity, The processor can compute, with the processor, arepresentative position value and a representative indicia value for aplurality of bins, each of the plurality of bins corresponding to one ormore of the mapped pixels and a portion of the connection feature. Theprocessor can output the uniformly scaled grayscale image using therepresentative indicia values and outputting a height image using therepresentative position values.

In yet another illustrative embodiment, a system for measuringdisplacement of an object surface comprises a displacement sensor thatprojects a line on the object surface and receives light from theprojected line at an imager in a manner that defines laser line positiondata and laser line intensity data. A vision system processor can beconstructed and arranged to generate a height image from the laser lineposition data, a grayscale image from the laser line position data andthe laser line intensity data, and an estimate of the grayscale imagegradient, the grayscale image gradient being used to compute a heightcorrection factor to be applied to each pixel of the height image.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention description below refers to the accompanying drawings, ofwhich:

FIG. 1 is a schematic perspective view of a laser displacement sensorsystem acquiring an image of an object as relative motion occurstherebetween in a scanning direction according to an illustrativeembodiment;

FIG. 2 is a schematic diagram of a portion of an imager pixel arrayincluding regions containing portions of an exemplary, imaged laser lineaccording to the embodiment of FIG. 1;

FIG. 3 is a schematic diagram of the imager of FIG. 1 showing thegeneration of a height image from an acquired image of a laser linewithin the displacement sensor's field of view;

FIG. 4 is a flow diagram of a procedure for processing height image datawithin regions of interest of the overall acquired image according to anillustrative embodiment;

FIG. 5 is a diagram of an image of a laser line in which two regions ofinterest are adjacent to each other, free of a gap therebetween;

FIG. 6 is a screen display of a Graphical User Interface (GUI) for usewith the displacement sensor system of FIG. 1, in which regions ofinterest for detection on the overall image of scanned object can be setin terms of physical units;

FIG. 7 is a diagram of a screen display including a GUI in which theuser manipulates regions of interest and exclusion on an image of theobject acquired by the displacement sensor;

FIG. 8 is a diagram of portions of an imager at two regions of interestshowing pixel rows that are selectively switched on or off to vary theresolution of the image at the region;

FIG. 8A is a diagram of portions of an imager at two regions ofinterest, in which the first region is at full resolution, with allpixel rows switched on, while the second row is at half resolution withevery other pixel row switched;

FIG. 9 is a diagram of an image of an exemplary object containing bothheight-varying features and variations in contrast from which aconcurrent height image and grayscale image can be acquired using thedisplacement sensor system of FIG. 1;

FIG. 10 is a diagram of a height image of the exemplary object of FIG. 9depicting variation in height of certain features as an intensityreading and also showing contrast-based false height readings;

FIG. 11 is a diagram of a grayscale image of the exemplary object ofFIG. 9 showing variation in contrast of certain features as a variationin grayscale intensity and depicting shadowing from certain raisedfeatures at height transitions thereof;

FIG. 12 is a flow diagram of a procedure for generating a height imageand a grayscale image from image data acquired using a scanned image ofthe displacement sensor of FIG. 1;

FIG. 12A is a flow diagram of a method for correcting for zero or moresuspect pixels in a row of the grayscale image possibly containingsuspect pixels;

FIG. 12B is a flow diagram of a method for resampling the acquiredimages, or the laser line position data and laser line intensity data,to generate a grayscale image with a uniform pixel scale;

FIG. 12C is a diagram of laser line position data and laser lineintensity data;

FIG. 12D is a diagram of mapped pixel values corresponding to the laserline position data;

FIG. 12E is a diagram of mapped pixel values including connectionfeatures;

FIG. 12F is a diagram of mapped pixel values including representativeposition values and representative indicia values;

FIG. 12G is a diagram of a portion of the calibrated height image andthe uniformly scaled grayscale image;

FIG. 13 is a flow diagram showing the compensation for contrast-basedfalse height readings in the height image using a comparison betweenfeatures of the height image versus corresponding features in thegrayscale image; and

FIG. 14 is a block diagram of a binary technique for determining acenter of the laser line in each column by comparing an intensity ofeach column pixel to a reference intensity.

DETAILED DESCRIPTION I. System Overview

FIG. 1 shows a vision system arrangement 100 including a laserdisplacement sensor assembly 110 oriented to image an object 120 (alsosometimes referred to as a “part”). The displacement sensor assembly(or, simply “displacement sensor”) can be contained in a single housing112 that is mounted at an appropriate location with respect to theimaged scene. In alternate embodiments, the displacement sensor cancomprise discrete, separated subcomponents. The object 120 and thedisplacement sensor 110 are in relative motion (double arrow My) witheither the displacement sensor 110, the object 120, or both, moving (thescan motion direction) along at least one axis. The motion direction isoften aligned with the depicted, relative (e.g. global) coordinatesystem 124 (in this example, the physical y-axis direction). In atypical arrangement, the object 120 is located on a moving surface 130(shown in phantom), such as a conveyor belt or motion stage, whichprovides motion data from an encoder or similar device operativelyconnected to the moving surface's drive system 132 in a manner clear tothose of skill in the art. The object 120 is, by way of example, astructure having a surrounding surface 140 that is relatively planar,with a height H1, and extends to an outer perimeter 142. Locatedinternal of the exemplary outer perimeter 142 is a surface 144 thatvaries in displacement along the physical z-axis direction between aheight H2 and a height H3. Note that each height, H1, H2 and H3 can beexpressed in physical dimensions, such as millimeters, from a base planethat can be the plane of the moving surface 130, or another referenceplane). Note again that this depicted object 120 is only one example ofa wide range of possible surface shapes, and is described as a usefulexample of the principles of the illustrative embodiment of the systemand method. In further examples the outer, surrounding surface candefine a more complex perimeter with additional features, such asfastener holes and the inner structure can be a more variable shape.Likewise, an outer, surrounding surface can be omitted form an exemplaryobject, in whole or in part.

The displacement sensor assembly 110 includes a laser assembly 150 thatincludes a lens or other optical arrangement 152 (e.g. a Powell lens ora cylindrical lens) that projects a curtain or fan 154 of laser light,toward the object 120 in a manner that forms a line 156 along thephysical x-axis (largely transverse to the scan motion direction My).The fan 154 resides generally in a plane that, in this embodiment, isapproximately (or substantially) perpendicular to a reference plane,such as the physical x-y plane of the moving surface 130 and/or a planeof the surrounding object surface 140. The displacement sensor 110 alsoincludes a camera assembly (dashed box 160) consisting generally of alens/optics assembly illustratively arranged in a well-known Scheimpflugconfiguration 162 and an imager assembly (“imager”) 164. The imager 164contains an array of photosensitive pixels at a predeterminedresolution, and the camera assembly generally defines an optical axis OAthat falls within the field of view where the line 156 is visible uponthe object at various surface heights. The optical axis OA defines anon-parallel (e.g. acute) angle Θ with respect to the plane (andassociated line 170) of the laser fan 154. Thus, any physical z-axisheight variation/displacement in the line along the physical x-axis isimaged by the camera assembly 160 as a variation in position (along theimage axis YI) in the resulting line 171 of a displayed two-dimensional(2D) image 172 as shown generally in the exemplary (and optional)display 174 that can be part of a computing device (e.g. a handheld,laptop, tablet or PC). As described further below, the computing devicecan interact with hardware and/or software that includes a graphicaluser interface process 176 and device setup process, each of which allowthe user to interact with the vision system processor 180 to set up,monitor and control the various functions and parameters of thedisplacement sensor 110.

Note that the depicted arrangement of axes (x, y, z) 124 is a conventionand other representations of relative orientation (e.g. polarcoordinates) are expressly contemplated. As shown, rotation about agiven axis is also represented, for example by the double-curved arrowRx, which depicts rotation about the physical x-axis.

The imager 164 is operatively interconnected to a vision systemprocessor 180 that is contained within the housing 112 of thedisplacement sensor 110 in this embodiment. In alternate embodiments,some or all of the vision system processor components and/or processescan reside remote from the housing (on an interconnected PC, forexample). The vision system processor 180 performs a displacement sensorprocess 182 in accordance with the illustrative embodiment described indetail further below. The processor 180 also receives relative motionsignals from the moving surface 130 or another motion-inducing device(e.g. a gantry or manipulator holding the displacement sensor and/or theobject). Acquired images are acquired in association with such motionsignals so that the line 156 is imaged at specified object positioning'sin the scene. As used herein, the term acquired images refers to acollection of pixel values on the imager. As also used herein and aswill be described in greater detail below, the term height image (orrange image) refers to the collection of height pixel values generatedfrom one or more acquired images and the term grayscale image refers tothe collection of grayscale pixel values generated from one or moreacquired images.

As also described below, the image data (i.e. a 2D height image—alsooften termed a “range image”) for the object is transmitted todownstream data handling devices and processes (190 in FIG. 1) for usein various production activities, such as quality control, objectmanipulation, assembly, etc. Users typically desire measurementscharacterized in physical units (e.g., millimeters or microns in x and yand z), rather than image units (i.e. pixels/pels in x, encodermotion-based triggers in y, and row displacements in z). For thisreason, displacement sensors are typically calibrated to convert betweenimage pixel outputs and physical units, such that the calibrateddisplacement sensor output is in physical units. Displacement sensorcalibration can be performed by the manufacturer such that thedisplacement sensor provides output in physical units (such as the modelDS1100 available from Cognex Corporation of Natick, Mass.).Alternatively, sensors can provide output in uncalibrated image units.When the sensor output is in uncalibrated image units, the user orapplication is tasked with handling input data in image pixel units. Theuser can also specify the desired height dimensions in terms of heightimage pixel scale (in physical units) and height image pixel-level scale(in terms of how many physical units each pixel-level represents) whenthe height image uses integral pixel values. In another example, theheight image can be represented as a 2D array of floating point values.

Note also, as used herein the terms “process” and/or “processor” shouldbe taken broadly to include a variety of electronic hardware and/orsoftware based functions and components. Moreover, a depicted process orprocessor can be combined with other processes and/or processors ordivided into various sub-processes or processors. Such sub-processesand/or sub-processors can be variously combined according to embodimentsherein. Likewise, it is expressly contemplated that any function,process and/or processor herein can be implemented using electronichardware, software consisting of a non-transitory computer-readablemedium of program instructions, or a combination of hardware andsoftware.

The sensor process 182 includes calibration values that are establishedat setup, which map the location of the line within the imager's fieldof view with a physical height (e.g. H1, H2, H3, etc.). The calibrationprocess can employ a calibration object that is moved along the z-axisvia a motion stage, or other conveyance to precise, known heights, andthe relative y-position (i.e. Y1 in the 2D display) of the imaged lineat that physical height is used to generate a curve or table of valuesthat reference each y-position with a given physical height. Eachy-position in each column (as noted below, there can exist multipley-values per column) is mapped to a physical height, and the samey-position in different columns can map to a different physical heightbased upon calibration parameters, and the like. Alternatively, valuescan be generated using a parametric model of the relationship betweenpixel locations and the x, z positions in physical units.Illustratively, the sensor process 182 determines the position of theline (e.g. the center or other specified location(s), such as the startand end locations of the detected line thickness), and then uses thisvalue in determining the physical height of that portion of the objectsurface. More particularly, and with reference to the diagram of FIG. 2,a portion of the imager 164 is shown. The imager consists of anM-row×N-column array of photosensitive elements commonly termed “pixels”210. These pixels each generate a charge based upon incident lightexposure. That charge is directed through an analog-to-digital converterarrangement (not-shown), where it is translated into a digital numericalvalue relative to the level of charge. This value is stored as imagepixel data for use by the processor 180. In a non-limiting example, apixel array of 768 rows×1024 columns can be employed. By way of furthernon-limiting example a field of view of approximately 175 millimeters isprovided with a vertical working range (WR) varying (along 170 from thelaser 150) of between approximately 140 millimeters (WRmin) and 315millimeters (WRmax). These dimensions are only exemplary of a wide rangeof operating parameters and are provided by way of further illustrationof the principles herein.

The depicted portion of the pixel array contains segments 220, 230 ofthe imaged laser line where the height transitions between the lower andhigher surfaces 140 and 144, respectively. That is, the pixels in thearray which are exposed to the light of the line are shown within thedashed lines of the segments 220, 230. Note that several rows fallwithin each line segment. The vision sensor process 182 typicallycomputes a line center in each column. This line center can be resolvedto a sub-pixel level based upon a variety of factors and computationsknown to those in the art. For example, resolution of 1/16 of a pixelcan be obtained using appropriate algorithms and the data from one ormore scans.

Each scan produces one line of a height or range image. As shown in FIG.3, the acquired image 310 of the field of view containing a laser line312 is processed to generate a N-wide field 320 of values 322 contain aphysical height (z-height, e.g. in millimeters) for each pixel column (0. . . N−1) of the acquired image. While in some embodiments, thephysical height for each value 322 can be generated at this time, thepresent embodiment defers generation of physical height until a furtherdownstream process step. In the present embodiment, the location of thelaser line in subpixel resolution is generated in pixel units, and at asubsequent step, calibration parameters are applied to the pixel unitsto generate a set of x, z values (i.e. the scan line in physical units).Note that there may be more or less (x, z) pairs in the physical unitscan line than there are in the line location (in pixels) vectorgenerated by the imager, because the scan line in physical units can beremapped to include finer/coarser sampling, as desired. Also, the numberof values (322) per column can vary along the X-direction so that morethan one value per column is generated. For example the start and endpositions of the laser line in the x-direction can be provided, as wellas the center and other metrics, such as the standard deviation. Ingeneral, the field 320 represents a height image consisting of heightvalues for each x-coordinate (column) across the scanned object surface.As the object is scanned, a series of successive fields can be combined(with knowledge of the y-direction motion between scans based on theencoder) to derive an overall profile of the object surface.

In generating each height image (320), the process reduces noise andimproves the accuracy in locating the line center position (and thusheight value) by applying a filtering algorithm to a neighborhood ofrows. A variety of conventional and/or novel filters can be employed. Inan illustrative embodiment, a line finding algorithm runs on the resultof applying a symmetrical 1-2-4-2-1 vertical filter kernel to theacquired image about each line for which a measurement is made, i.e. forevery row in the acquired image pixel values in current row aremultiplied by 1, the pixel values from the previous row multiplied by 2,the row before is multiplied by 4, the row before that is multiplied by2, and the row before that by 1. The filter output is the sum of thefive products listed above. Choice of the 1-2-4-2-1 filter is odd-sizedso that the center application point of the filter can correspond to thesame location as the result of the application of the filter. Otherfilters may be used based on user preference, project parameters, orother circumstances. For example, such other filters may includedifferent widths and/or different coefficients, such as 1-2-4-6-4-2-1 or1-3-3-1, etc. The filter values in rows above, at, and below the maximumfilter response location are used to determine line location withsub-pixel accuracy using quadratic interpolation (or other knowntechniques). If there is a plateau in filter response, then the centerof the plateau is used instead of the peak position.

Special consideration should be given to the rows close to the beginningand the end of the region in the image. For the first and last lines,the system does not generate a filter measurement. For the second andsecond-to-last lines, the system approximates the desired 1-2-4-2-1kernel with a 3-4-2-1 or 1-2-4-3 kernel, respectively. In anillustrative embodiment, the line finding process can, moreparticularly, employ the following filters relative to various locationsin the region of interest (also described further below) of the image:

Row number (in the analyzed region) Filter used 0 No filter measurementmade 1 3-4-2-1 2, 3, . . . 1-2-4-2-1 2^(nd) last row 1-2-4-3 Last row Nofilter measurement madeAgain, note that the filter response for a given row cannot actually becomputed until the two following rows have been acquired—except just asingle following row is used when computing the response for thesecond-to-last row in a region. In addition, when the user indicates theheight measurement should encompass the range Hmin-Hmax in physicalunits, we need to convert that to a range of rows Rmin to Rmax and thenadd at least one row each at top and bottom such that we can have filteroutputs that correspond to measurements in the full Hmin to Hmax range.

II. Split Regions of Interest

With reference to FIG. 2, it is contemplated that various surfaceprofiles can contain features of interest that are separated by asignificant distance within the working range, and features outside ofthe regions containing features of interest are lacking or unimportantto the analysis of the object. In the example of the object 120, thebase surface 140 and raised surface 144 are of interest and nosignificant features exist in the height range between these surfaces.Thus, a region of interest exists around height H1 and between heightsH2 and H3. All other heights are unneeded for the analysis. However,significant processing time and resources can be spent in analyzing theentire field of view for line information. In many applications, thegeneral location of a feature of interest within the field of view canbe predictably determined, within a relatively small height range.According to an illustrative embodiment, and with further reference tothe procedure 400 of FIG. 4, the sensor process defines a plurality ofregions of interest ROI1 and ROI2 (FIG. 2) in the field of view. Eachregion includes a plurality of rows in the imager. Each region ofinterest is sufficiently tall in the column dimension of the imager toaccommodate the expected location of a laser line based upon the heightof the object features and variations in height due to motion of theobject relative to the field of view, or due to other processrequirements. In an embodiment, each region of interest row includespixels from all contiguous columns. According to the procedure 400, theuser, based upon an estimate of the physical height of object featuresof interest, their variation in height, as those features move throughthe scan and variations in the process, establishes regions of interestthat the process coordinates with pixel locations in the imager (step410). At the appropriate encoder (or other motion-based) value, theimager acquires an acquired image of the object (step 420). The processof setting regions of interest causes the imager, or another downstreamcomponent to acquire image pixel data exclusively (only) within thepixel rows residing within the set regions of interest. As such, theprocessor can focus resources on image data only from each region ofinterest, thereby reducing image readout time (as image pixels outsidethe regions of interest are not transmitted) and speeding thecomputation of surface height in each image. Note that the region ofinterest for the line segment 230 is sufficiently large to accommodatethe variation in surface height of the surface 144 as the object movesthrough the scan along the y-axis as well as other process requirements.

In an embodiment, the imager includes a capability to, via the processor180, switch on or switch off the output of selected rows. In alternateembodiments, the imager can set ranges of rows to switch on (allowingreadout of selected rows) or off (preventing readout of deselectedrows). Other arrangements that block the processing of predeterminedrows of image data can be implemented in further embodiments.

In step 430, the procedure 400 employs the above-described line findingalgorithm to locate the position of the line with sub-pixel accuracy.Laser line intensity data and laser line position data can also becomputed by peak detection. For each column, the peak value/position ofthe filter response, e.g., the acquired pixel values subjected to thefiltering process above, is found. The laser line position data, whichcomprises one or more laser line position datum, corresponds to theposition of the filter peak value (in each column). The laser lineintensity data, which comprises one or more laser line intensity datum,corresponds to that filter peak value. The term datum is used to referto a measured value, such as a pixel value; the line intensity datumrefers to the laser line intensity pixel value, and the line positiondatum refers to the laser line position. The term data is sometimes usedto refer to a set of individual datum, and sometimes used to refer to anindividual datum. The position of the line in each column is used instep 440 to construct a slice of an overall height image representingthe profile of that slice of the object. In optional decision step 450,the measurements of a slice can be analyzed to determine if acondition—such as an out-of-specification object is present. If so, thesystem signals the condition (e.g. a failure) in step 452 and takesother appropriate action (e.g. ejecting the failed object, sounding analarm, stopping the line, etc.), and can loop to inspect the next objectin step 410. However, it is expressly contemplated that variousapplications can continue to scan an object even after an erroneouscondition is identified (via dashed branch 454, which returns to scanacquisition step 420). Note also that the step 450 can also be used tocount single or multiple scans (slices) of the image. This/theseslice(s) can represent a region of interest, or combination of regionsof interest, in the field of view. That is, while some system processesemploy a scan the entire object surface (for example, to ensure thatalignment, measurements, and decision-making are based on the full 3Drepresentation of the object), some system processes can desirablyemploy individual scans (slices) of the object surface (i.e. partcross-sections), and e.g. align, measure, yield data and/or makedecisions based upon each individual scan/slice. Thus the slice istransmitted to one or more downstream processes via step 456.Additionally, some system processes use groups of scans (e.g. tencross-sections at a time), and combine this data using an appropriatetechnique (such as smoothing, averaging, other image processingoperations, etc.). This combined data is then used to align, measure,yield data and/or make decisions. Scanning can continue as this group isprocessed via branch 454 as slices are sent to the downstream systemprocess(es) in step 456. Moreover, some system processes employ acombination of above to provide fast measurements/decisions, etc. basedon individual/groups of scans, and final measurements, etc. based on theentire (or a larger portion of the) scanned object.

Absent a signaled condition (fault, single slice, multiple slices) instep 450, the scan can continue (via decision step 460) until images ofthe line located over the entire object surface (or a desired portion ofthe surface) have been acquired. The individual height images are thenused to construct a 3D profile of the object surface for use bydownstream data handling processes (190 in FIG. 1), such as qualitycontrol, defect detection, part ejection, etc. Notably, downstreammeasurements or similar information that encompass physical datagenerated by the analysis of more than one region of interest can beused in one or more decision-making process(es) (192 in FIG. 1) and/orvision system task(s) (194 in FIG. 1). It is also contemplated that thedownstream measurements or similar information can employ physical databetween more than one ROI generated by the same scan. More particularly,various image processing operations undertaken on the regions ofinterest can be applied across (and applying data from) multiple regionsof interest in the overall height image to derive particular informationabout the object. Such processes can include, but are not limited topattern-matching operations, inspection operations, pick-and-place (andother robot manipulator) operations, tolerancing operations, and/or anyother vision system function that can employ image data derived from aheight image. The process continues (via decision step 480 for eachobject until done (step 490).

With reference to FIG. 5, in certain implementations the features ofinterest can be located in closer height proximity such that the fieldof view 510 contains a line that is analyzed using two adjacent orcloser proximity (and non-overlapping) regions of interest ROI1 a andROI2 a. In this illustrative arrangement, the above-describedline-finding and filter process (530) is modified to accommodate thelack of gap between the regions. In this special case, the beginningportion of the above-described table is modified to accommodate thesecond region. Note that regions can generally be arranged in a mannerthat the first row in the second region is the next to the last row inthe first image, i.e. there is no gap between them. Thus, the filterscan be applied to each adjacent region (ROI1 a and ROI2 a, respectively)applied in the following manner:

First Region:

Row number Filter used 0 No filter measurement made 1 3-4-2-1 2, 3, upto 2^(nd) to last row 1-2-4-2-1 Last row No filter measurementSince the sensor acquires only every other row in the second region thenarrower filter used in the first region cannot be run in the secondregion. However the wider filter that is used throughout the secondregion can also be applied in the last few rows of the first region. Thewider filter results in this region overlap the results from the narrowfilter and can be used to “blend” data between the regions.

First Region Overlap Area:

Row number Filter used Last - 3 row 1-2-4-2-1 Last - 1 row 1-2-4-2-1

Second Region:

Row number Filter used 0, 1, onward 1-2-4-2-1 Last - 1 row 1-2-4-3 Lastrow No filter measurement

Notably, a GUI process (176 in FIG. 1) that is instantiated in theprocessor 180, the attached computing device, or both, enables the userto select regions of interest (typically) during setup. FIG. 6 shows aGUI display 610 of the setup interface for the interconnecteddisplacement sensor 110. The display 610 includes a plurality of tabs620 that access respective screens with which to train the sensor andset various parameters. The selected tab 622 displays the “profile”screen 630 that includes input boxes with which the user can select“Both” (632) to define multiple (two in this embodiment) sensed regionsof interest in the overall detection base. The “Both” selectionillustratively provides two columns 634 and 636, associated with eachrespective region of interest to be established in the field of view.Each region contains a setting for the physical location (inmillimeters) of the Z Detection Base 640 (the start of the detectionregion from the z-origin of the working range), the Z Detection Height642 (the z-distance of the respective region of interest) and a Zdetection sampling value. In this example, the first region Z Detectionbase begins at 30 millimeters above the origin and extends for a heightof 10 millimeters. The second region's Z Detection Base begins at 79millimeters and extends for 14 millimeters thereabove. Both regions aresampled at a value of 1. Any areas in the field of view outside theseregions are excluded from processing. Note that in embodiments in whichmore than two regions of interest are to be established, the “Both”selection can be substituted with a different dialog box, thatestablishes the number of regions and associated columns can beemployed. To assist the user's understanding, these relationships (foreach region of interest) are selectively depicted in a graphic 650 shownat the side of the input area of the display 630. This interface canallow generation of multiple regions of interest.

FIG. 7 depicts an alternate implementation of a GUI for setting updetection and exclusion regions. In this embodiment, the display 710includes a displayed image 720 of the entire field of view of the object730. A laser line 740 is shown projected over the object 730. The usercan move the object (via the conveyor, motion stage, etc.) to determinethe limits of movement of the line in each region and the boundaries ofassociated regions of interest ROI1 b and ROI2 b are selected by theuser. One or more appropriate buttons 760 can be employed to set regionsof interest, in association with cursor (750) movements or similarinterface functions known to those of skill. Having graphically set theregions of interest, the setup process translates these locations intophysical measurements and locations along pixel columns in the imager.

III. Resolution Adjustment

Note, in further embodiments, as described below. Some or all of therows can be defined at a reduced resolution—for example sampling everyother row. In addition, and as shown in the example of FIG. 2, the moredistant line segment 220 appears at a lower resolution, defining a widthLW1, crossing fewer pixel rows than the closer line segment 230 whichdefines a width LW2, crossing more pixel rows that the line segment 220.Thus, the closer line segment is subjected to a denser scan and higherresolution than the further line segment. In general, the closer thedistance of the imaged light within the working range, the higher theresolution of the image. Typically, the resolution of the further linesegment is generally sufficient. Illustratively, the resolution ofregions of interest at a shorter working range can be reduced byselectively switching off the output and/or processing of rows. Forexample in the region of interest, ROI2, every alternating row, can beswitched off. For further resolution reduction every third or fourth rowcan be switched on, while intervening rows are switched off. Theresolution of a longer-working-range region (ROI1) can also define areduced resolution as appropriate in accordance with this technique.Where the resolution of both regions is reduced, a greater reduction istypically applied to the region at a shorter working range to compensatefor its initial high resolution. Note that the ability to varyresolution can also be used to address the existence of a narrowed“waist” at the central region of the imaged laser line, and widened endson each side of the waist.

By way of further illustration, FIG. 8 depicts an exemplary portion ofan imager 800 including the pixel rows 810 associated with two regionsof interest ROI1 c and ROI2 c. As shown, output instructions 810 areissued to the imager control circuitry 830 and/or to the vision systemprocessor (based upon an automated arrangement or a GUI setting, forexample) to discard signals (readout) from every alternating row(switched-off rows being designated by an “X”) in the first region ROI1c. Likewise, in the second region ROI2 c, every third row is switched onwhile the two intervening rows are switched off. In general, by reducingresolution, an acceptable profile is computed, while reducing processorworkload and increasing the throughput speed of the scan. Moreparticularly, the illustrative embodiment shown in FIG. 8A provides animager 850 that transmits full resolution in the first region ROI1 d(switched-on, readout of every row) and half resolution in the secondregion ROI2 d (switched-on, readout of every other row). Controlfunctions and instructions are similar to those described with referenceto FIG. 8 above.

IV. Generation of Grayscale Images

In many manufacturing applications, among other processes, it isdesirable to inspect both the height profile of an object, as well ascertain 2D elements, such as printing. For example, the keys of akeyboard should all reside at a predetermined height and also eachpresent the proper printed indicia (alphanumeric characters, symbols,etc.).

By way of example, FIG. 9 is an image 910 of a common object (a licenseplate) 920, which contains both flat (relative to the surrounding platebase 930) printed elements 940 and raised elements. While each set ofelements can be provided in a contrasting color (e.g. blue and red), itis contemplated that the laser color can be selected to appropriatelyresolve both colors in an acquired image. Based upon the acquired imagedata, the system can measure the maximum filter response of the acquiredimage intensities after each column is subjected to the filteringprocesses described above, such as a 1-2-4-2-1 filter, to create agrayscale image that is as insensitive as possible to height changes andshows contrast changes on the surface, such as printing. Alternatively,the system can use the same quadratic or other interpolation function tocreate an estimate of the response at the measured line subpixelposition, and use that to create the grayscale image. Thus, the systemcan generate either a height image, a grayscale image, or bothsimultaneously (interleaved images). In some examples, the grayscaleimage can be generated from filtered laser line intensity data and thefiltered laser line intensity images may be greater than an 8-bit value.This is because the 1-2-4-2-1 filter applied to an 8 bit value can yielda result greater than 8 bits. Since some machine vision routines expectan 8 bit grayscale image input, the system 100 can map laser lineintensity 16-bit values to 8-bit grayscale image values. The user canemploy a pixel map to map from the filtered laser line intensity image,which can have a bit value greater than 8-bits, to an 8-bit grayscaleimage. The grayscale image data can be post-processed according to thecorresponding height information because laser lines may be thinner aslaser return may be thinner and laser return may be dimmer the furtherthe laser lines are positioned from the sensor. The bit-mappingdescribed above can be application specification and/or part specific.

FIG. 10 details an associated height image 1010 of the object 920. Notethat the height image 1010 is represented (for the purposes ofillustration) with varying intensity based upon height, with black 1030representing the height of the baseline of the working range, gray 1040representing the elevated height (from baseline) of the plate base 930and white 1050 representing the height of the raised elements 950. Theflat printed elements 940 are also generally grey, matching the heightof the base plate 930, but, as described below, include “false” edges1060 that appear as slight height changes.

FIG. 11 represents a grayscale image 1110 of the object 920 acquiredusing the image data derived from the scan of the object underillumination of the laser line. As shown, the contrasting colors of theflat elements 940 are imaged clearly relative to the contrastingbackground of the plate base 930. The raised elements 950 are alsoimaged (although the color may be washed-out if a similar color laser isused—selection of an appropriate color and/or optical filter can ensurea complete grayscale image of the object is acquired). The raisedelements may also contain subtle features 1150 due to shadowing from theangled perimeters of the raised elements. This effect can besignificantly reduced by applying corrections based on grayscale image.For example, the vertical gradient can be calculated in each pixel ofthe grayscale image and correction based on this gradient can be appliedto each height measurement in the height image.

As shown in the diagram of FIG. 12, the scan of the laser line generatesan acquired image 1210. This acquired image is subjected to line-findingprocesses/algorithms 1220, as described generally above. Theseprocesses/algorithms 1220 can act upon a region of interest or theentire image as appropriate. The image data 1230, consisting of pixelgrayscale intensity values in each column and the peak intensitylocation (and intensity value) are transferred by the line-findingprocess to be used in constructing a slice of a height image 1240, andalso a slice of a grayscale image 1250. In another example, the imagedata 1230 can include grayscale intensity values that have beengenerated by subjecting the laser line intensity data to one or morefilters described above. The height image uses the peak location in eachcolumn to generate a physical height value, while the intensity at peakis used as a grayscale value for that x-y location in the image.

As the image is scanned and a plurality of scans (1210) are acquired,the process generates further height and grayscale slices (1240, 1250)to add to overall height and/or grayscale images 1260, 1270 of theobject. As described above, that the user can direct (via the GUI, forexample) that either type of image, or both types of images, begenerated. In an embodiment, one group of pixel rows can be used forgenerating height images and another group of pixel rows (e.g.alternating pixel rows, etc., as shown generally in FIG. 8) can be usedto generate grayscale images. In this manner, the resolution of eachimage can be reduced by using fewer pixel rows, but discrete (differing)rows of pixels in the imager can be used concurrently to generateside-by-side height and grayscale images.

In some examples, the generated grayscale image data can include peakintensity datum values that are a result of a bad pixel location in theacquired images. As described above, the appearance of suspect or badpixels in the grayscale image may be a manifestation of a defect orirregularity in the hardware or software used to generate the grayscaleimage. In particular, the use of imaging components with bad sensingelements can cause the appearance of such suspect or bad pixels. It canalso cause incorrect results in the height images and in the grayscaleimages. It is therefore desirable to ignore the data corresponding tobad sensing elements by identifying suspect peak data values and usinginterpolated grayscale values from nearby data in place of the suspectpeak data values. In another example, anomalous height values cansignify that the associated grayscale information is suspect, e.g.,suspect grayscale image values can be identified by analyzing theassociated height information, either in the laser line information orin the height/grey level information.

FIG. 12A is a flow diagram of a method 1200A for correcting for zero ormore suspect pixels in a row of the grayscale image possibly containingsuspect pixel datum values according to the description above. As anexample, the method 1200A may be instantiated at any component withinthe system, such as the vision system processor 180, sensor process 182,or data handling device/processes 190. The blocks of method 1200A,although depicted below in a certain order, can occur or be performed byone or more processors and in any particular order.

At block 1210A, the laser line peak position data and the correspondinglaser line peak intensity data are received or generated. The laser linepeak position data and corresponding laser line peak intensity data canbe generated as an output of block 430, as described above. The laserline position data and laser line intensity data can each respectivelyinclude one or more individual laser line position datum and laser lineintensity datum, each having a respective value. At block 1220A, thepeak intensity location in the laser line peak intensity data may beidentified, as also described above.

At block 1230A, each datum value of the laser line position andintensity data is compared to a suspect pixel map to identify one ormore suspect or bad datum values in the laser line intensity data thatcorresponds to the laser line position data. The location of suspect orbad pixels can be known prior to the imaging process. In this regard,the suspect pixel map can be developed at block 1235A during thecalibration process for the imager. The suspect pixel map canpre-identify particular pixels of an imaging device that are known to besuspect or bad pixels. The suspect pixel map can be generated at anytime during the process, and in particular can be generated duringcalibration of the imager. The suspect pixel map can be generated byacquiring one or more images of uniformly bright or dark scenes during acalibration imaging process and identifying pixels that measures valuessignificantly different from nearby sensing elements. Alternatively oradditionally can be generated using known suspect pixel informationprovided by the manufacturer.

At block 1240A, a comparison is made between the peak intensity locationidentified at block 1220A and the location of any suspect pixellocations. If the peak intensity pixel location is identified thatcorresponds to, overlaps with, or is within a predetermined proximity of(e.g., within a few pixels), a suspect pixel location, the system candetermine that a corrective action should be undertaken to correct thesuspect datum value in the laser line intensity data. One suchcorrective action may include, for example, generating a substituteintensity pixel value at block 1250A and substituting the originalintensity datum value corresponding to a suspect pixel location in theimager with the substitute intensity pixel value in the laser lineintensity data at block 1260A. The term original value refers to theintensity value provided by the sensor, or processing steps, prior toperforming blocks 1250A and/or 1260A, and the term substituted valuerefers to a new intensity value provided by block 1250A. The substitutepixel value may be computed according to any number of methods. In oneexample, the substitute pixel value may be computed by taking averagevalues of the intensity of adjacent grayscale pixels, such as agrayscale pixel to the left and a grayscale pixel to the right of thebad pixel. In another example, the substitute pixel value may be apredetermined constant pixel value. In yet another example, aneighborhood-based operation may be performed on surrounding pixelvalues to determine the substitute pixel value. Such neighborhood-basedoperation may include any type of calculation performed on pixel valueswithin a predetermined range of the bad pixel. As discussed above, oncethe substitute pixel value is determined, the substitute pixel value maybe substituted in for the value of the original datum value associatedwith the bad pixel location on the imager. Once substituted, a correctedgrayscale pixel for the laser line peak intensity data can be outputtedat block 1270A.

If the peak intensity pixel location in the grayscale image does notcorrespond to, does not overlap with, or is not within a predeterminedproximity of (e.g., within a few pixels), a datum value associated witha suspect pixel location, the system can output the grayscale pixelwithout correction at block 1245A. In another example, the system canalternatively undertake corrective action as shown at blocks1250A-1270A, despite there being no overlap or correspondence among thepeak intensity pixel location and the suspect pixel location.

In another illustrative embodiment, suspect pixel values can beidentified from the corresponding height image. For example, if aparticular height value differs from its expected height value in theheight image by a predetermined threshold, then the corresponding pixelin the grayscale image can be labeled as a suspect pixel. Thepredetermined threshold of difference between the expected and observedheight values can be defined by a user. In another example, thepredetermined threshold can be computed as a difference between adjacentpixels.

In another illustrative embodiment, suspect pixel values can beidentified from the corresponding height image by pixels for which theheight value is missing. For example, if a particular height value ismissing from the height image due to laser speckle or occlusion oranother cause, then the corresponding pixel in the grayscale image canbe labeled as a suspect pixel.

The grayscale image data can be used in a variety of vision systemprocesses such as inspection, registration, print quality analysis, andany other process in which the presence, absence location and/or qualityof contrasting elements is analyzed. Notably, the illustrative systememploys the same imager, and concurrently acquired image data from theimager, to derive both the height image and grayscale image, so theseimages are in essentially perfect registration—that is each image pixelof the height image corresponds with the same pixel in the grayscaleimage. This eliminates the need for a separate camera assemblies toacquire each type of image and the associated need for an alignment stepto correspond the features of the height image and grayscale image. Theexistence of two physically aligned images with different types of data(height and grayscale) allows for application of a variety of visionsystem tools with both types of data in concert. This increases theusefulness of the overall image data set, allowing for such concurrentoperations as registration, pattern matching, etc. using the grayscaleimage and measurement, inspection, etc. using the height image. Certainsimilar vision system processes (e.g. inspection) can also be applied toboth images to provide a higher level of confidence in the result. Forexample, a hole is located in the grayscale image and the position ofthe corresponding height change in the aligned height image is used toverify its location. More generally, the system can provide ameasurement process that employs (a) measurements of data from theheight image to construct measurements of the grayscale image or (b)measurements of the grayscale image to construct measurements of theheight image. A decision-making process (192 in FIG. 1) then generatesdecisions based upon the measurement process. A variety of decisions canbe made in this and other embodiments herein including, but not limitedto, object rejection, line stoppage, alarms and alerts

V. Resampling of Grayscale Image to Adjust for Pixel Scale

In some examples, the grayscale image and the height image can havedifferent characteristics that can prevent the useful comparison of thetwo, or can prevent their simultaneous use in a machine vision task. Asdiscussed above, the grayscale image may be distorted or may have adifferent pixel scale than the height image. To correct for suchdifferences, the vision system can resample the underlying acquiredimages, or the laser line position data, laser line intensity data, orbad-pixel-corrected intensity data (as described above), to generate auniform pixel scaled grayscale image with respect to the height image.

FIG. 12B is a flow diagram of a method 1200B of resampling the acquiredimages, or the laser line position data and laser line intensity data togenerate a grayscale image with a uniform pixel scale. The resamplingmethod 1200B can be performed as a pre-processing step at the visionsystem processor 180, or more particularly, the sensor process 182. Theblocks of method 1200B, although depicted below in a certain order, canoccur or be performed by one or more processors and in any particularorder.

At block 1210B, laser line position data 1210C and laser line intensitydata 1220C are generated by the system 100 from acquired images. Inparticular, the laser line position data 1210C and laser line intensitydata 1220C can be generated from the acquired images at the visionsystem processor 180 or the sensor process 182, and may be the output ofblock 430, as described above, which outputs the laser line intensitydata corresponding to the laser line position data. As shown in FIG.12C, both the laser line position data 1210C and the laser lineintensity data 1220C can each be arranged as a row of pixels, but anyconfiguration or orientation of the pixels is contemplated. The row oflaser line position data 1210C can include a plurality of pixels,arranged in a row, that correspond to a row of the acquired imagesacquired by the imager 164 and that has been appended with peakintensity values. The laser line intensity data may include a pluralityof pixels, arranged in a row, which corresponds to height image datathat has been subject to one or more filters, as has been describedabove. In one example, the laser line intensity data can correspond tothe output of the method 1200A, as also described above. The laser lineintensity data 1220C may also include indicia 1230C indicative of albedovalues measured at each pixel. For example, an albedo valuecorresponding to the color blue can be used to indicate a darkergrayscale value measured in the laser line intensity data generated fromthe acquired images. Similarly, an albedo value corresponding to thecolor red can be used to indicate a brighter grayscale value measured inthe laser line intensity data generated from the acquired images. Inother examples, other albedo values or colors may be used to signifyparticular grayscale values in the laser line intensity data.

At block 1220B, each pixel of the row of the laser line position data1210C is mapped to a corresponding coordinate (x,z) in physical space.In one example, and as shown in FIG. 12D, the mapping can include plot1240C, which includes the plotted pixels 1250C. Each of the plottedpixels 1250C includes a z coordinate that corresponds to the value ofthe laser line position data 1210C. In particular, the pixels of thelaser line position data 1210C that have a darker shade of graycorrespond to a greater z value. More particularly, the calibrationprocesses described above provide a mapping from imager coordinates tophysical coordinates. The calibration is used to map from (x,y) imagerpixels positions (x=column and y=row) to physical positions (x,z in mm).The laser line position data characterizes the found laser line peakimager y position (row) for each imager x position (column). Thecalibration is used to map the values in 1210C to the plotted pixels1250C. In this example, the leftmost pixel of the laser line positiondata 1210C corresponds to the leftmost plotted pixel 1250C, as do theremainder of the pixels continuing to the right in plotted pixels. Inother examples, there may be a pixel offset in the x direction.

At block 1230B, as shown in FIG. 12E, connection features 1260C areconstructed to connect the adjacent plotted pixels 1250C. In oneexample, the connection features 1260C can include line segments, andcan include similar indicia 1270C corresponding to the indicia 1230Cdescribed with respect to the laser line intensity data 1220C.Similarly, the indicia 1270C may have an albedo value corresponding tothe albedo value set forth in the laser line intensity data 1220C. Thecorresponding albedo values allow the appropriate color to be drawn intothe uniform pixel scaled grayscale image at the corresponding location.

At block 1240B, as shown at FIG. 12F, for each bin 1280C having a widthparallel to the x-axis, representative position values or range valuesare computed from the z coordinate values falling within the bin. In oneexample, the representative position values or range values can includean average. The width of the bin can be determined by any number offactors, including the desired width and resolution of the resultinguniform pixel scaled grayscale image. In this regard, each of the bins1280C may correspond to zero or more values from the laser line positiondata 1210C. Each of the values that fall within each respective bin1280C are averaged to compute an average position value or range valuefor each bin 1280C. In another example, the computer average positionvalue can be computed from the average range values of the connectionfeatures 1260C within each bin.

For each bin 1280C, a representative indicia or albedo value is alsocomputed. In one example, the representative indicia or albedo value caninclude an average. As discussed above, the connection features 1260Care labeled according to indicia corresponding to certain albedo values.For each bin 1280C, the albedo values that fall within the bin are alsoaveraged to yield an average albedo value for each bin. In anotherexample, the computed average albedo value can be computed from theaverage albedo values of the connection features within each bin.

At block 1250B, as shown at FIG. 12G, the output 1290C corresponds tothe calibrated height image in block 440, which is output in uniformpixel scale. The calibrated height image corresponds to the averageposition value or average range value computed above. A uniform pixelscaled grayscale image 1295C is also outputted. The uniform pixel scaledgrayscale image 1295C includes indicia 1297C corresponding to theaverage indicia or albedo values previously computed. For example, theindicia 1297C may include the color purple where one of the binsincludes a connection feature 1260C that is part red and part blue, withpurple being the average of purple and blue. The outputs 1290C and 1295Care now resampled to have the same pixel scale allowing for apixel-to-pixel comparison to be made among the outputs. This has manyadvantages and allows for various machine vision tasks to be performedon the images. In one illustrative embodiment, the resampled grayscaleimage can be generated at different (x,y) scales than the height image.This can be accomplished by performing the various blocks set forth inthe method 1200B twice to provide a different x scale. The grayscaleimage can further be resampled in y to provide a different y scale.

VI. Compensation for False Height Transitions

As shown in the height image 1010 (FIG. 10) above, contrast changes showup as slight false contours (i.e. elements 1060) in the height image.The height measurement is not affected significantly by the absolutealbedo of the surface in an area of constant albedo, and thus thecentral region of each element 940 exhibits approximately the sameheight reading as the surrounding plate base 930. However, at the edgeof the feature 940, when the light-to-dark or dark-to-light transitionis passing through the measurement zone, there is an error induced inheight measurement.

FIG. 13 shows a generalized procedure 1300 that compensates for suchfalse height transitions. Height image data 1310 and grayscale imagedata 1320 are each registered (step 1330) so that height correctionfactors can be generated at each point in the image in step 1340 by theprocessor. Illustratively, the procedure 1300 operates continuously,using an estimate of the grayscale image gradient in the scan directionto compute the height correction factor(s) that is/are applied at eachpoint in the height image (step 1350). Illustratively, the correctionfactor can provide a numerical value—for example where zero (0) is nocorrection to a point in the height image. A zero, or near-zero,correction factor is applied in regions of the height image where thegrayscale contrast is not changing. For example, the correction factoris minimal or zero at points residing remote from (and internal of orexternal of) the edges 1060 of FIG. 10, while a more-significantcorrection factor is applied at points lying on the edges due to thesignificant contrast gradient. By applying such correction factors, acorrected/compensated height image is generated in step 1360. Thisheight image omits most or all of the effects of contrast-based falseheight readings. A variety of techniques can be employed in furtherembodiments to compensate for false height and other contrast-basedeffects.

In determining whether false height and/or grayscale features arepresent, the setup and training of the system can includeuser-identification of candidate false-features and/or regionscontaining candidate features. This can speed the analysis of suchfeatures and improve throughput of the displacement sensor.

It is expressly contemplated that a runtime (or setup) display (FIG. 6)can be provided to a user showing the above-described height image,grayscale image and corrected/compensated image selectively or incombination(s). Using custom and conventional GUI techniques, the usercan be provided with various functions that allow for adjustment ofvarious image parameters and/or characteristics. For example, buttonsand setting inputs can be provided on a GUI to enable the user to setand/or adjust thresholds and other image parameters (e.g. sensitivity,display contrast, etc.) based upon the displayed height image, grayscaleimage and/or corrected/compensated image.

VII. Line-Finding from Column Threshold Pixel Voltages

It is contemplated that processing speed can be increased in variousimager arrangements by finding the line center value for each pixelcolumn directly from “binary” pixels that either exceed or fall below athreshold. In this manner, each pixel is delivered to downstreamprocesses/processors as either a logical 1 value (e.g. indicatingpresence of the laser line) or a 0 value indicating absence of the lineat that location in that imager pixel column. FIG. 14, shows anarrangement 1400 in which the imager 1410 outputs intensity values 1412from the pixels to a circuit (e.g. a comparator circuit) that compares(1420) the intensity of each pixel in a column to reference intensity1422, and determines which pixels exceed the threshold value establishedby the reference intensity. The column locations of the first and lastpixels in each column that exceed the threshold value (and have a 1value), and are generally likely to constitute a laser line aredetermined in block 1430. Note that other pixels in the column at aspacing from a laser line may return a 1 value due to noise,specularity, etc., but the process should filter such pixels as notconstituting a group sufficiently large and contiguous (thoughpotentially containing small gaps of 0-value pixels) to define a laserline. Note that the term “intensity” can be defined as a voltage valuein various embodiments. That is, while various commercially availableCMOS imagers convert the read-out pixel signal of each row fromintegrated irradiance, to charge, and then to voltage in the readoutprocess, it is contemplated that other imager types could potentiallyreadout directly in the charge domain directly, or as a current valueinstead of a voltage value. Thus, the term intensity is meant to defineany such signal.

Based upon the first and last pixel location in the column, the processdetermines the center value in block 1440. This can be accomplished by aspecialized hardware circuit or by transmitting the pixel valuelocations to the vision system processor. It is contemplated, that whilea thresholding process is used to locate line pixels in each column foruse in constructing the height image, the processor can also receive theintensity value of the pixels (for example in a region of interest) anduse such values to construct a slice of a grayscale image of the objectsurface.

Note that as used herein the generation of a “center” value can employ avariety of techniques that allow for an increase in processing speedversus accuracy—for example, the first value in a column to exceed thethreshold in a cluster of over-threshold pixels can be identified as the“center”. Thus the term “center”, should be taken broadly to includelocations along the column within the laser line image that return apredictable result. Illustratively, knowledge of the laser waist size(width) at a given column (x-location) can be used in conjunction withthe row location of the first over-threshold pixel to compute thecenter. For example (in pixels) the following simplified computation canbe employed:

Center Location=First Pixel Location+Waist Size/2

VIII. Conclusion

It should be clear that the various embodiments described hereinincrease the versatility, processing efficiency and throughput speed oflaser displacement sensor systems without sacrificing needed accuracy.The illustrative functions described herein can be readily implementedby a user during setup by employing appropriate interfaces, in astraightforward manner. These functions allow for correction of certaininherent errors in height images, such as contrast-based false heightreadings. Systems and methods are provide to produce registeredgrayscale images of an object under scan free of the use of a separatecamera system and associated processors.

The foregoing has been a detailed description of illustrativeembodiments of the invention. Various modifications and additions can bemade without departing from the spirit and scope of this invention.Features of each of the various embodiments described above may becombined with features of other described embodiments as appropriate inorder to provide a multiplicity of feature combinations in associatednew embodiments. Furthermore, while the foregoing describes a number ofseparate embodiments of the apparatus and method of the presentinvention, what has been described herein is merely illustrative of theapplication of the principles of the present invention. For example, asused herein various directional and orientational terms such as“vertical”, “horizontal”, “up”, “down”, “bottom”, “top”, “side”,“front”, “rear”, “left”, “right”, and the like, are used only asrelative conventions and not as absolute orientations with respect to afixed coordinate system, such as gravity. Also, while regions ofinterest are of constant size and location in the field of view/imagerthroughout the scan of each object, it is contemplated that the sizeand/or locations of regions of interest can be varied during the scan toaccommodate expected change in locations of feature or change infeatures of interest along the y-axis. Variation of the regions ofinterest can occur in response to encoder position or another indicationof the relative position of the object with respect to the field ofview. In addition, while the terms “row” and “column” are used todescribe a particular arrangement of pixels in the imager, it isexpressly contemplated that these terms can be interchanged and achievesimilar results in accordance with embodiments herein. Also, while alaser is used to generate illumination for acquisition of both theheight information/image and grayscale information/image, supplementalillumination can be applied in acquiring the grayscale image—forexample, alternating acquisition of the height image and the grayscaleimage and strobing the supplemental illumination with each grayscaleacquisition. Also, while embodiments herein define regions of interestin which processing of image pixels occurs and regions therebetween thatare free of processing of image pixels, such intervening regions can befunctionally equivalent to one or more “exclusion zones” in which theuser defines regions of non-processing, and the system otherwisedefaults to performing processing outside such exclusion zone(s). Moregenerally, in embodiments the system and methods define one or moreregions of interest and associated exclusion zone(s) allowing forselective detection or non-detection (respectively) of a laser linewithin the overall imager field of view. Accordingly, this descriptionis meant to be taken only by way of example, and not to otherwise limitthe scope of this invention.

What is claimed is:
 1. A system for measuring displacement of an objectsurface comprising: a displacement sensor that projects a line on theobject surface and receives light from the projected line at an imagerin a manner defines a plurality of displacement values in a heightdirection; and a vision system processor that processes rows of imagerpixels to determine a position of the projected line in columns of theimager pixels exclusively in each of a plurality of regions of interest,each of the regions of interest respectively defining a plurality ofrows of imager pixels that correspond with expected locations of theprojected line on the object surface, wherein of the position of theprojected line in each of the columns corresponds to a physicaldisplacement value on the object surface.